Aren't they suing him? Mm-hmm. Can't th can't they get legal aid? Hmm. Mm-hmm. Are we supposed to count to ten or something? We don't have to do all that? Okay. Okay. Thank you. Mm-hmm. Okay, so You're gonna give a presentation for us then. Great. Mm-hmm. Mm-hmm. I was just wondering if you were gonna define this more on the slides to come or if this is as much as you're gonna tell us. I'm just wondering if he's prepared Sure. So so what did you have more on later slides about this? Okay. But we've also established that it's not just speaker change, it because if you have mm-hmm, okay, then Okay. Y yeah, yeah. Right. Yeah. Mm-hmm. Well it might be the case that to be interactive there have to be dialogue acts of some specific type in a turn rather than Like statement, statement, reject, accept, you know, something like that. Mm-hmm. Well, actually I don't quite agree. I mean i for example you can't use R_S_T_ across speakers because the definitions of the relations depend on the intentions of the speaker and there's different speakers have different intentions. So you can't use the theory. It's just fundamentally wrong, it's it was meant as a theory for a one person uh right. So that doesn't mean there aren't other theo so for Yeah. Right, but those are very those are really different um theories that are at are trying to get at different things and it and they admit of m you know, some of them admit of multiple parties and some of them assume you're in one speaker's head. Right, and the Mm-hmm. Yeah, the entire theory is based on intention and the intention of one individual. So R_S_T_ would be a bad theory to use. W So modelling modelling a single contribution by a single speaker, you can within that you can use R_S_T_ if you want to, right. But across speakers you can't. They have no relations for things like reject or agree. They the theory doesn't have them. So the the so it's not the right theory 'cause the theory is the relations plus the notion of nuclearity. So we need a different theory. Now I think Alex um S_D_R_S_ allows both kinds. So you know, within a speaker turn you can have things like explanation and cause and all that. Um but then across speaker turns you can have things like correction or I didn't see any examples of agree or disagree in S_T_R_S_, but I there aren't? I mean there are thirty relations. So we should look at at what they are. But I d I don't Well her the relations have a s D Yes, but so I mean the relation names it's not the names that are important, it's their semantics that are important, right. And they're the semantics of the relations are based on the semantics of the utterances that they're relating as Alex s explained yesterday. Um and so um it is important that these relations have a semantics that's tied to the semantics of the utterances, right. Uh p poss possibly, but few people have tried to, you know, do the kind of um truth conditional semantics that, you know, Asher and Lascarides have done, right. So the theory can handle the kinds of things that happen in meetings, as she explained. The actual computational mechanisms they have currently for building up the trees can't. Now I don't remember, was which speaker you in the c in the example she showed yesterday, was it the case that the representation of the utterances and the relations included which speaker was uh I mean it had the at the l at the leaves they had the well, she didn't show it, but they would have the semantics of the utterances. And then she had relation names and handles. Right. I just I just don't didn't remember seeing in the representation which speaker contributed what. 'Cause she did represent um discourse structures 'cause these were the Verbmobil scheduling dialogues. So the utterances in the in the representation were uttered by two different parties that wasn't the you know, in the in the examples she showed us, there were there was some that was just, you know, one party. It was a monologue, but then there was a snippet of dialogue as well with For this specific set of relations that that they've defined. There might be some higher level structure that you wanna pull out of that. I mean I 'cause you were saying you're just gonna you you're you wanna look at this, you know, very local stuff. But from that you c one would hope you would pull out something high, you know No, I think working from the bottom up sounds great, but I'm just y I mean what you can imagine someone saying well how did they decide to make it blue, and maybe there was a long blow by bl and you don't you kind of wanna abstract away from he said this, she said that, he sai you know, to Anyway, I you know, I mean I I don't think especially uh, you know, if you're not steeped in these theories that it makes a huge difference to you whether we use R_S_T_ or something like S_D_R_S_. But it might make much more, you know, I mean Well I think we should ass I think we should try to find a theory that allows multiple speakers and relations to span over more than one spea you know. Because it is true that I mean if you look at things like damsel and other attempts to come up with annotation schemes for for dialogue acts, there was this notion of forward looking and backward looking. Anyway, they're relational. I mean in some way, you know, they're all relational. Um but you needed But there's a different set of tags in there from the kinds of relations you see within R_S_T_, for example. So if you look at Amanda Stent's thesis and the analysis that she did sort of within a turn she used something like R_S_T_. And then across turns she used damsel, right. Well I Well, it gives you things like accept and reject and Oh yeah. In fact R_S_T_ doesn't give you anything like that. Okay. Right. Right. Right, okay. D so I think it's much more impor I think it would make more sense to go about this problem figuring out what kinds of relationships we want and then finding and, you know, like setting down what the constraints are on the analysis game. He's been looking at the data, okay, s to good. Um and then and then there are lots of different theories of argument. I mean you can also think about the sort of joint problem solving kinds of um approaches. So you know, Clark and Lockbaum and, you know, that that kind of thing. That were designed for dialogue, right, I mean, and can be expanded to more than two persons. Actually the Lockbaum stuff is for groups. Yeah, I was just gonna say it would be useful to know what m But who's they? That's the question. Who? Right. Mm-hmm. Well I mean I think that um w it I don't know uh w it'd probably get to re hear, you know, more of your presentation because I think there are lots of different candidates and you certainly can't go through and mark up every meeting, I mean mark up meetings with every scheme we think of. So I think we have to kind of get an overview of what the candidates are m if there's like two or three that we think are good contenders, then we try to see what works better, right. There are a lot of option there are a lot more options. So so let's see what what else you've got. Oh, so that was it on dialogue, okay. Okay. So uh it's very unclear to me why you think exchange of information is more principal t uh did you I mean what did you have specific schemes in mind there? Or spec mm specific people's work that you um But can you give us an example of a scheme that's based on exchange of information? Okay. Mm-hmm. Okay, so I mean I think that um there are a lot of things that we have to look at before, I mean Yeah. I mean and Well, we're marking up a lot of higher level things in our Okay. So um have you looked at Nate Blaylock's work? So he has things like, you know, adopt um so basically it's to, you know, collaborative problem solving, you know, more than one person involved, and trying to characterise things as um You know, if I I can't remember the I can't remember the names of all the actions. No, so so the Coconut project is a project that I did with Rich Thomasson and Barbara DiGinio. Have you looked at that? Not very clear. No. Though there's a lot of detail in the paper. But you haven't looked at it no in very much detail. Okay, so I think you should, um because it was all these things that are done for joint problem solving and joint action I think are very relevant to look at because these people are solving a problem together, right. So um while these schemes may not have they have some es of the essential components that you need to characterise um joint problem solving. So while th um you may think that they're missing something, then we we could certainly augment them. But I think that they have to be looked at. And so so Nate Blaylock tried to okay, so you know, from Gross and Krauss and Gross and Sidener came all this stuff about shared plans and how you really can't understand, you know, dialogue u unless you understand shared plans. But the actual theoretical um or formalisation of it was incredibly complex. And so um Lockbaum simplified it quite and, you know, Sidener and Rich and Collagen have simplified it quite a bit. Um but Nate Blaylock tried to um, you know, define, you know, what happens after somebody suggests something. Well, you can adopt it, you can well I can't remember you know, okay, but he has uh he has a taxonomy yeah, right. Yeah, yeah. Yeah, exactly, right. Um which we tried to apply to tutoring, but it actually doesn't work very well for tutoring 'cause tutoring's just a totally different ball game. But um it you know, we spent a fair amount of time looking at it and it did strike me that for joint problem solving there w there were some good ideas. And there is, you know, a set of tags and what they mean and, you know, that sort of thing. And then you can also look at, you know, cl you know Clark on joint action. Now he's not gonna give you a set of tags 'cause that's not the kind of guy he is. But um But the overarching idea, you know, I mean I think accounts for a lot of the ways in which um, you know, these other people are thi are thinking. But I mean I think that we we basically have to merge some of this stuff that's been done, you know, on meetings with some of the stuff that's been done on collaborative problem solving 'cause there's elements of each that seem really important. And I think th Because it really seems to me that, you know, if you look at and some of this stuff on exchange is like ancient and you know, they didn't have the ability to mark up lots of data and count things, you know. So I mean all the conversational analysis stuff, I mean those guys never count anything, you know. I mean they're uh ps so they see a pattern and then they think it's kinda cute and then they scour the world for third turn repairs, you know. They don't actually say how often do those happen. I mean they worked I mean I know Manny Schegloff really well and they're not counters. They're purposely not counters. They define themselves as non-counters. Mm-hmm. Yeah, th I mean it it does seem like, you know, there's something that we want from that and then there's something that we want from collaborative problem solving. And if you l if you take the exchange stuff, it's not gonna d quite do everything you want, and if you look at the collaborative problem solving stuff, maybe it won't do every that you want. And so Hmm. Like somebody's writing on the white-board or I mean the w meeting actions they presented at the review seemed really low level to me, like somebody stood up. Okay. Okay. Okay. Okay. I mean I I realise it's important to recognise that, you know, what Wei-Xun got up and went to the white-board. I realise that's important, you know. But I d was kind of surprised that those meeting actions. So Yeah, mm okay. Well I think we should approach it both ways. I mean uh you know, I I d You know, the meeting browser people have a lot of constraints. And so what they end up with saying is most critical for them right now might not be the most interesting thing in the longer term. And so Right. I mean people have said over and over again that they would like to understand how a decision has evolved, right. Um and I think that Well, I don't know if all of this low level stuff would allow you to track that kind of thing, but it might at least be the building blocks of allowing you to track that kind of thing and Yeah, some higher level yeah. So I mean I think it's clear that something at this level needs to be there. I mean and to ask questions, like who disagreed or were there any counter proposals or whatever. Even those kind of questions you need some kind of mark-up like this. Well that brings up something interesting because I think probably what we should do is try to get you know, some of these tasks sort of put on, you know, sort of as part of the official agenda. So being able to answer questions like um were there any counter proposals to the bana you know, banana shape or whatever. Um uh or to some specific, you know, recommendation and things like that. I mean, you know, those, you know, strike me as cl clear things that, you know, people who go to decision making meetings want to know if they've missed one or are arriving late or anything like that. So it wou it seems like we should probably get some of these things on the official agenda and say look, you know, wouldn't it be nice if you could have answers to these questions, or something. Oh. No, m So And I think we wouldn't wanna necessarily do this on ICSI data. I think the ICSI data is too too complex, and there aren't enough real decisions made, you know, that you could clearly identify. I mean there are some, but and not that I've read more than about two meetings. But um I think, w it, you know w well, so one thing that's gonna be interesting is the, I think to me, is the um what can you do because presumably the um D_F_K_I_ folk or somebody is gonna mark up a lot of the data with sort of tags about ontology, right? So we know, you know, somebody will have gone and by hand and sort of marked up som something l like shallow semanticy kinds of things, right, or maybe deeper semanticy things, but that but you know, not really in a in a formal language, but mostly just, you know, tags, right? Um and then the question is, well, how does you know, just the way people typically do this is, you know, let's take these semantic features into or these features into account and see how well we do with the the learning, and then let's throw these features out and see how well we do, um because we know we're gonna have them, because one or two sides is really intr are really interested in this ontology stuff, right? I mean T_N_O_ is and D_F_K_I_ is, right. So there will be some kind of domain semantics. And there will also be presumably some kind of task hierarchy modelling going on in I mean 'cause the whole browser working group um final um sort of presentation on that second day was about, you know, knowing the task, right. And very you know, details about which task is this you know, which sub-task is this little chunk of the meeting addressing and furthering in various ways, right. Which seems like it would be really hard in an open domain. You remember when Steve came in, Steve Whittaker came in at the end and said, you know, they'd spent the morning talking and they came up with this idea. Um No, they said people are talking about tasks, you know. I mean they really they made it clear that they were that that was gonna be heavily No, those are the s those are the scenarios. Those are the user scenarios. The task what what he came back and said that at the end, I mean it actually kind kind of scared me, because it was all based around tasks. And and the domain tasks, okay, they're now they now have to decide how many buttons to put on this thing. I mean it was all about the domain task. And it is really the it and given that, uh it it's really scary to think about what you could possibly do in an open domain where you don't where you don't know what the tasks are. And I don't think you're gonna be able to figure them out uh very reliably. I do, but the whole project is, you know, focussed on AMI hub data as their first priority. N no no no no no, but I mean if the if the if they had th they had this sort of epiphany at the end of this meeting where they decided well why are people browsing this. Well it's all to see how, you know, how the participants were furthering their tasks. Well right. And then, yes, and then there was this confusion about the um last spider phone conference that the messages telling people to call came out after the meeting happened, at least to some of us. So um I mean we'll we'll get that back on track. But I think that um we don't you know, there will be some meeting browser demo that um, you know I I think it's an interesting question of okay well, if you have knowledge, here's what you can do. If you don't have that knowledge you have to do different things and, you know, you can't necessarily Well I think when he does re when he does lear these learning experiments, he should do 'em two ways. Because he will have this opportunity with or without. And then we can see like well without, you know, these tags so you know, Alex was showing um yesterday well without knowing the head word of the utterance at the leaf, you know, you get these kinds of results, and when you add that in, you get these other kinds of results. And you know, I think yeah. Um so I think if we have those features, it's, you know, it's a rare orp or rare opportunity that you can, you know, you can try one head to head against the other. Um and then of course to make sure that you really um Right. Well we won't I mean we won't use 'em till they're ready obviously. So we might do the sort of open domain, I mean a although it will be a closed domain, I mean we we can't use any domain information necessarily. Uh I mean, there probably are a bunch of issues like if you've got a hundred hours of meetings and they all talk about a lot of the same things, then it's ea it it's not really open domain and somehow that sneaks into all the statistics anyway 'cause you don't have such a wide vocabulary and, you know Right. Sure, so we might find that when we move from what we thought was open domain, but on the hub, you know, our results go down when we truly are getting meeting yeah. So I mean that'll be interesting. Okay, well that's good. So okay. So we wanna decide um what Wei-Xun should do next and maybe think about meeting in, I don't know, a month or something. Yeah, mm-hmm. Um so before we do that, can I just ask you a question about chunking? In the beginning you said something about you were doing this chunking um and you're doing it on the ICSI data, but this was did you say it th wasn't picked up in AMI or you I I I was confused about w Mm-hmm. Mm-hmm. And you and you haven't done it yet. Tha that's fine. So that's still on the agenda to do at some point. Yeah. Exactly. That's Well so I mean right. Um or building blocks. I mean so I mean we have to think about how s chunking interacts with the automatic segmentation, you know. Because um Barbara was showing the low level. And I mean it seems to me that chunking should have some kind of so if if your dialogue act segmenter says there's there's a segment here, but your chunker said no, this is a coherent chunk, then y y you know, w I mean maybe they could work together to do a better job than Right. Yeah. Periods. Oh, after uh right. Okay. Right, okay. Good. So it sounds like well w Well how much AMI Well, you wanted to do the chunking on the AMI data, right? Because m So we got stuck at some point on evaluation of chunking, right. The problem was you didn't have ICSI data that was hand labelled for chunks. And we decided labelling the ICSI data for chunks would be probably difficult and maybe not very useful. And so it it mi Okay, if there if there was if there was hand labelled data that you could evaluate again so train on some and evaluate on some, wouldn't that um 'cause you've trained on Switchboard, right, where there there are chunks. So if you had some ICSI data that you could maybe use, I don't know, um as a what's the n there's t there's a training set and then there's the development set and then there's the test set. Maybe you'd use some chunked ICSI stuff as development or something and do some last tweaks and then evaluate. But you know, w h I mean I can see why Wei-Xun didn't wanna spend his time chunking ICSI d Which means we have to get some portion of the data hand-chunked. How much um well how much in Switchboard. Switchboard is all chunked. Right. Right. Well No, you had Penn tree-bank parses, and you made chunks from that. Mm-hmm. Is anybody gonna parse? Is any gonna try to anybody try to parse the AMI data? Well they're using the E_R_G_, right? The which probably has a l maybe finer g uh which I believe has finer grained distinctions than, and different labels, than the Penn tree-bank. It's I'm t it's probably easy to dumb down the H_P_S_G_ parse too. 'Cause the Penn tree-bank parsers are pretty minimal because they didn't wanna commit to a specific grammatical theory. So I mean it probably is worth looking at 'Cause they're doing this with Dan Flickinger, right? Which is the same um parser as the Redwoods Tree-bank was made with that Alex talked about yesterday. So you can look at Redwoods parses um and see how different they are from tree-bank parses. And you might wanna talk to Alex to find out if somebody's already done this turn Redwoods stuff into tree-bank stuff so that they can compare, you know, against results that are reported on tree-bank. Well that was the idea I believe, but um it was supposed to be another tree-bank. Well the good thing about that is that then you know that when you go through and parse you'll be getting the same chunks as when you do this process automatically. 'Cause the problem w when you go through and hand do it is then suddenly you introduce your automatic method and your accuracy goes can go way down. I mean sometimes it doesn't if you know, sort of for part of speech tagging, you know, whether you do it by hand or machine, it's pretty much the same accuracy, right. But They talked about it at the meeting that we went to in Twente. I was thinking Enschede. Why can't I remember the n the f the right name of this place? Yes, Twente. Which You were there Jean. Yeah, okay. No no no no, no no no, I I I know you missed a meeting and s and so there there may have been you missed the one in T_N_O_. On on which project? Sorry, I didn't get that. Verbmobil, right. Yes, yes, they did. It well that's that's what Alex talked about yesterday, right. The parse I mean that's wasn't something she did, but that's the data she's using, right. So we have all that stuff here, you know, the parsed Verbmobil material. So you can look at that and see what the parse trees look like. You don't need to wait for D_F_K_I_. And if it turns out that you think oh, there's no way I can map these into Penn tree-bank style bracketing, then then we have to think of something else, right. But I think they had that in mind. So it's it's worth also talking to her about that. Okay, good. Everybody always says there's no data like more data. You know, I mean the typical thing is to do leave one out cross validation with, you know, eighty nine dialogues. Okay, so I trained on eighty eight and then or I train on one thousand nine hundred and ninety nine and Okay, but we should speak to Steve. We might have some um Well the question the question is what what is chunking gonna help? I mean I think chunking could help automatic dialogue act segmentation. I think chunking could help, as you mentioned, named entity recognition. Um Mm-hmm. Mm-hmm. Well we'd wanna make sure we're pretty happy that it's reliable because Mm yeah. I mean otherwise you get these very bizarre results because the data is not really very good. So um but I and nobody has expressed like massive interest in t in chunking. So I think, you know, nobody's uh waiting for it. But but we could we c I think we should talk to Barbara about how to do some joint experiments or or if she's gonna be sending other people, maybe Liz or whoever comes over next time. Uh and it would actually be great if it was Liz, because I think she is the one who did the dialogue act segmentation automatic She just said that hopefully by the next meeting she would be bringing someone else, or someone else would be coming instead of her, remember? The th not to have not to have it done, but just to talk about how it could interact with automatic dialogue act segmentation. Um and we I guess we should talk to Yoshi about about it in relationship to named entities. Who's uh who's doing named entities besides Yoshi? Okay. So we could talk to Vasilis about that as well. And then So one thing that would be very helpful for me because I haven't read I think we're in a situation where Jean, you know, has read a lot of literature and I've read a lot of literature and we there is some overlap, but there's also quite a big space where we don't know the literature from the other side. And since you're gonna be reading it, I mean it would be probably be useful if you made some kind of, you know, report that described, you know, okay, this what these people you know, this is, you know, a synthesis okay, here's the basic idea behind exchange structure and, you know, here's a couple of you know, here's the two most relevant works that I've read or something like that, you know, to just Or even just like a paragraph of, you know, the basic idea. Just because, you know, obviously I'm not gonna go rea off and read all this kind of stuff because I don't have time. And you probably don't have time to read all this collaborative prol problem solving. So Well in any way, every time you write a paper, you've got to go back and do this related work thing anyway. So it just it just makes it a lot easier um to get that common ground I think, you know. I think it's a real advantage that we've read different things and come at it from a different side because I think this ha they the the previous approaches haven't really, you know, taken you know, the good features from, you know, both sides and and tried to put them together. Yeah, me too. I can send you a l Well I think we both wanna send w well, I don't know, maybe you've already done this, but I you know, I will send you or we can, you know, chat off-line for a minute a about a list of things that I think you really need to take a look at. And Jean, probably maybe you should do the same. Um And if Sabrina has any ideas, we'd be happy to hear from Sabrina as well. Um Mm-hmm. Okay. Yeah. Mm-hmm. Mm-hmm. M Mm-hmm. Hmm. Yeah. Mm-hmm. Well I think we need to um, like you mentioned earlier, um factor in what this might be used for in the end. And so even if it we won't it won't be used, maybe if it would need some higher level analysis before it could be directly used for that purpose, we should still keep that in mind. And so I think going through the presentations that um were given at the work package six meeting where um and I think it was um it wasn't wha was it Anita or was it oh I've forgotten her name, the other woman um from Twente, you know, they talked about they did they had done a lot of questionnaires, and they got some user requirements out of that. And then Steve presented um the results of following people around and looking at how they used their notes from meetings. And and so that would give us some ideas about, you know, what people want, you know, out of meeting records. But then we should also add things that we think are interesting. But just to have to keep some of these goals in mind 'cause they're t o one scheme might let you answer this set of questions but wouldn't help you at all with another set of questions. So should we expect that all the slides from like the work package six meeting or the joint work package five work pa will be u up on one of the work package web sites? And if not, how do we make that happen? Hmm okay. Some of this was presented in at least the one spider phone or the second spider phone telecon I went to for work package six. So if they're not on the web we can try to look at that, you know, at the materials that were presented in that telecon without necessarily listening to the whole telecon. Yeah? Wh And Steve m and his I've forgotten the name, is it was it Simon, his yeah. They they've I think they may have written this up um in a report um, but I wou would hope that's on the work package six web site as well. I do as i once he's read a couple more I mean he's just completely missed out the collaborative problem solving world, which I d I really think we need to take into account. I mean you may take it into account and say no, you're wrong, and here's why, but I I I'm gonna need some convincing. Or t or or two and see this one gives you this and this g and can't you know, are they fundamentally incompatible. Or can we yeah, mm-hmm. Okay. I think this is gonna take about three weeks 'cause he's got plenty of reading to do. So maybe we get together again in the yeah, I mean, I'll send them the references immediately. Is that enough time for you do you think? Is that enough time for you to to read and think about If y if if it's too tight, you know, I think m we can say four. Or are you gonna be on holiday? Yeah. Maybe four weeks? Okay. Okay. So we should get together probably in about four weeks, yes. So when are you gonna bring this beautiful baby in to see everybody? Good, good. That's great. Yep. Well good luck solving your problem. I have to tick and tick? It does something weird when you do that.